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^ ■ 

\^ i The model for homogeneity of proportions in a two-way contingency-table/cross- 

tabulation is the same as the model of independence, except that the probabilistic 
■ process generating the data is viewed as fixing the column totals (but not the row 

totals). When gauging the consistency of observed data with the assumption of inde- 
pendence, recent work has illustrated that the Euclidean/Probenius/Hilbert-Schmidt 
' distance is often far more statistically powerful than the classical statistics such as x^i 

c/2 . the log-likelihood-ratio G^, the Freeman- Tukey/Hellinger distance, and other mem- 

bers of the Cressie-Read power-divergence family. The present paper indicates that 
the Euclidean/Frobenius/Hilbert-Schmidt distance can be more powerful for gauging 
the consistency of observed data with the assumption of homogeneity, too. 

, Keywords: chi-square. Fisher's exact. Freeman- Tukey, likelihood ratio, power diver- 

gence, root-mean-square 
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^ 1 Introduction 



The statistical analysis of categorical data is commonly formulated in the framework of 
^ I contingency-tables/cr oss-tabulations; Table [J provides a typical two-way example (see, for 
instance. Chapter 4 of Andersenl . 1990i for a comprehensive treatment). A common task is to 



ascertain whether the given data (displayed in Tabled]) is consistent up to expected statistical 
fluctuations with the model for homogeneity of proportions (displayed in Table |2]). When 
considering homogeneity, we assume that the probabilistic process generating the given data 
fixes the column totals (but not the row totals) by construction. Therefore, to gauge whether 
the given data displayed in Tabled] is consistent with the assumed homogeneity displayed in 
Table 12] we do the following: 

1. We generate s sets of draws, with the kth set consisting of n. ^ independent and 
identically distributed draws from the probability distribution {pi,p2, . . . ,Pr), where 
Pi = ni^Jn, p2 = n2^./n, . . . , Pr = rij-^Jn. Note that Pj = [rij^, ■ n_^k/n)/n_^k for j = 1? 2, 
. . . , r; these are homogeneous proportions (since pi, p2, ■ ■ ■ , Pr are the same for every 
column index k). 
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Table 1: A typical two-way contingency-table/cross-tabulation [rij^k is a nonnegative integer 
with j = 1, 2, . . . , r, k = 1, 2, . . . , s] rij^, = J2k=i ^ total with j = 1, 2, . . . , r; 

n.^k = Y7j=i ^j,k is a column total with A; = 1, 2, . . . , s; and n,^, = Y7j=i Sfc=i ^j,k = n is the 
grand total) 
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Table 2: The model for homogeneity of proportions n2^., . . . , n^^. are the row totals; 
^^.,1; "^.,2; • • • ; '^.,s ^re the column totals; and n.^. = n is the grand total) 
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2. For each of the s sets of draws — say the fcth set — we define Nj^k to be the number 
of draws falling in the jth row, for j = 1, 2, . . . , r. 

3. We calculate the probability P that the discrepancy between the simulated counts Nj^k 
and the model iVj . ■ N_^k/n is greater than or equal to the discrepancy between the 
observed counts nj ^ and the assumed rij . ■ n^^i^/n. When calculating this probability, 
we view Nj^k and iVj . as random, while viewing all other numbers as fixed. Please note 
that, by construction, A^.^^ = n.^fc for A; = 1, 2, . . . , s. 

The number P defined in Step 3 is known as the (exact) P-value. Given the P-value 
P, we can have 100(1 — P)% confidence that the observed draws are not consi s tent w ith 



assuming the homogeneity displayed in Table 121 See Section 3 of iPerkins et al.l f|2011l) for 
further discussion of P-values and their interpretation; Section 3 of iPerkins et al.l (120111 ) 
details subtleties involved in the definition and interpretation of these P-values. 

The definition above of the P-value P requires a metric for measuring the discrepancies. 
The canonical choices are and the log-likelihood-ratio G^: 
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r s 

j=i k=i 



(iV,,, - (iV,- . ■ N,,/n)f 
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(1) 

(2) 
(3) 

(4) 
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Pg2 = Prob{G^ > g'} (6) 

Other possibilities include the Hellinger (or Freeman- Tukey) distance and the Frobenius (or 
Hilbert-Schmidt or Euclidean) distance: 



j=i k=i 



= 4 E E( - V ■ N,k/nf 

j=i k=i 

Ph2 = FToh{H^ > h'^} 



(7) 



(9) 



j=i k=i 

r s 

j=i k=i 

Pp = ProbjF^ > f} 



(10) 



(11) 
(12) 

When taking probabilities in ([2]), (E]), dl]), and ( IT^ . we view the uppercase X^, G^, i/^, and 
as random variables, while vie wing the low ercase x^y 9^) and p as fixed numbers. 
As discussed, for example, by iRad (120021 ). X^, G^, and all converge to the same 
distribution in the limit of large numbers of draws — X^, G^, and are the best-known 
members of the Cressie-Read power-divergence family. is not a member of the Cressie- 
Read power-diver gence family and doe s not necessarily converge to the same distribution as 
X^, G^, and H^. iPerkins et al.l (120111 ) illustrated the many advantages of when neither 
the row totals nor the column totals are fixed; the present paper illustrates the advantages 
when the column totals are fixed. However, F^ is not uniformly more powerful than the 
classical statistics. We recommend using both F^ and a classical statistic such as G^. 

In the sequel. Section [2] summarizes an algorithm for computing the P-values defined 
above. Section [3] analyzes several data sets. Section H] draws some conclusions. 
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2 Computation of P- values 



The definitions of tlie P- values in ([3]), ([6]), (|9]), and ( IT2|) involve the probabilities of certain 
events. In the present paper, we compute these probabilities via Monte-Carlo simulations 
with guaranteed error bounds. Specifically, we conduct a large number m of simulations; in 
each simulation — say the ith — we perform the following steps (using the data of Tabled]): 

1. We generate s sets of draws, with the kth set consisting of n,^k independent and 
identically distributed draws from the probability distribution (pi,p2, • • • ,Pr), where 
Pi = ni^Jn, p2 = n2^./n, . . . , pr = Ur^./n. Note that pj = [rij^, ■ n.^fc/n)/n.^fc for j = 1, 2, 
. . . , r; these are homogeneous proportions (since pi, p2, . . . , Pr are the same for ev- 
ery column index k). Furthermore, the underlying distribution of the draws does not 
depend on i. 

2. For each of the s sets of draws — say the kth set — we define n^-^]. to be the number 



of draws falling in the jth row, for j = 1, 2, . . . , r. 

f^^^ between the simuiaucLi ^.wuhud /i,^-^ 



3. We calculate the discrepancy f?p-. between the simulated counts nfl and the model 



n^j'] ■ n[2/n, that is. 



r 



/(l = EE("ri-«'-"!2/"))^ (13) 

j=i k=i 



An estimate of the P- value P/2 is the fraction of f^^-^, f'^^y ■ ■ ■ , ffm) which are gr eater 



than or equal to p defined in ( ITOl) . As discussed in Section 3 of IPerkins et al.l (j201ll ). the 
standard error of the estimate is a/-P/2(1 — Py2)/m, where m is the number of simulations. 

Needless to say, we can compute the P- values for x^, 5'^, and h"^ via similar procedures, 
with the same error bounds. 

Remark 2.1. For all computations reported in the present paper, we generated r andom 



numbers via the C programming language procedure given on page 9 of iMarsaglial ( l2003l ). 
implementing the recommended complementary multiply with carry. 



3 Data analysis 

To compare the performance of the various metrics for measuring the discrepancies between 
observed and simulated data, we analyze several data sets. Using the procedure of Sec- 
tion [21 we conduct m = 4,000,000 Monte-Carlo simulations per P-value, for each of the 
examples presented below. The standar d error of the r esultin g estimate for the P-value P is 



then a/-P(1 — P)/2000; see Section 3 of IPerkins et al.l ( 2011 ). Before reporting the P- values 



associated with the data sets, we make two remarks concerning their interpretation: 

Remark 3.1. A significance test can only indicate that observed data cannothe reasonably 
assumed to have arisen from the model of homogeneous proportions; a significance test 
cannot prove that the observed data can be reasonably assumed to have arisen from the 
model of homogeneity. Thus, aside from considerations of multiple testing, if any statistic 
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.0868 
.0906 
.0959 
.0905 
.00838 



strongly signals that the data cannot be reasonably assumed to have arisen from the model 
of homogeneity, then we must reject (or at least question) the model — irrespective of any 
large P-values for other statistics. For instance, if the P-value for the Frobenius distance 
is very small, then we should not accept the model of homogeneity, not even if the P-values 
for x^, the log-likelihood-ratio g^, and the Freeman- Tukey/Hellinger-distance /i^ are large. 

Remark 3.2. The term "negative log-likelihood" used in the present section refers to the 
statistic that is simply the negative of the logarithm of the likelihood. The negative log- 
lik elihood is the same s tatist ic used in the generalization of Fisher's exact test discussed 
by lGuo and Thompson! (119921 ): unlike the log-likelihood-ratio G^, this statistic involves only 
one likelihood, not the ratio of two. We mention the negative log-likelihood just to facilitate 
comparisons; we are not asserting that the likelihood on its own (rather than in a ratio) is 
a good gauge of the relative sizes of deviations from a model. 

The 11 X 2 Table [3] displays the data for our first example, which has 22 entries in all. 
Table m displays the model of homogeneous proportions for Table [31 The P-values for Table [3] 
for the assumption that Table H] gives the correct underlying distribution are 

log-likelihood-ratio (G^) 
Freeman- Tukey/Hellinger {H"^) 
negative log-likelihood: 
Frobenius (F^) 

Please note that the P-value for the Frobenius distance is over an order of magnitude smaller 
than the P-values for the classical statistics. 

The 7x3 Table [7] displays the data for our second example, which has 21 entries in 
all. Table |H] displays the model of homogeneous proportions for Table [71 The P-values for 
Table [7] for the assumption that Table [H gives the correct underlying distribution are 

log-likelihood-ratio (G^) 
Freeman- Tukey/Hellinger (if^) 
negative log-likelihood: 
Frobenius (F^) 

Please note that the P-value for the Frobenius distance is over four times smaller than the 
P-values for the classical statistics. 

The 9x2 Table [Til displays the data for our third example, which has 18 entries in all. 
Table [121 displays the model of homogeneous proportions for Table [Til The P-values for 
Table [TH for the assumption that Table [T21 gives the correct underlying distribution are 

X'{X^)-- .123 

log-hkelihood-ratio (G^): .138 

Freeman- Tukey/Hellinger (i?^): .157 

negative log-likelihood: .114 

Frobenius (F^): .0344 

Please note that the P-value for the Frobenius distance is over three times smaller than the 
P-values for the classical statistics. 



.145 
.292 
.493 
.132 
.0286 
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The 5x3 Table [15] displays the data for our final example, which has 15 entries in all. 
Table [11] displays the model of homogeneous proportions for Table [IS The P-values for 
Table [TB] for the assumption that Table [TH] gives the correct underlying distribution are 

X^{X'): .276 
log-likelihood-ratio (G^): .171 
Freeman- Tukey/Hellinger {H"^)'- .0794 
negative log-likelihood: .235 
Frobenius (F^): .199 

In this example, none of the statistics produces a very small P-value; the smallest arises from 
the Freeman- Tukey/Hellinger distance in this case. 

Remark 3.3. Appropriate binning (or rebinning) to uniformize the frequencies associated 
with the entries in the contingency-tables/cross-tabulations can mitigate the problem with 
the classical statistics. Yet rebinning is a black art that is liable to improperly infiuence the 
result of a significance test, and the usual data- dependent rebinning calls for Monte-Carlo 
simulations to calculate P-values accurately anyways. Rebinning always requires careful 
extra work. A principal advantage of the Frobenius distance is that it does not require any 
rebinning; indeed, the Frobenius distance is most powerful without any rebinning. Note also 
that optimally rebinning data such as that displayed in Table [3] can be very challenging. 



4 Conclusion 

The Frobenius distance is significantly more powerful than the classical statistics for gauging 
the consistency of observed data with the assumption of homogeneity in many of the examples 
of the present paper. This may or may not be typical of most applications; actually, we 
suspect that our last example — in which all the statistics perform sir ailarly — is the most 



representative. Even so, both the present paper and the applications of [Perkins et al.l (l201l[ ) 
illustrate that there are many important circumstances in which the Frobenius distance is 
much more powerful than the classical statistics. 
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T able 3: Re s ults o f polls in June 1983 for Danish parliamentary elections, from Chapter 4 
of lAndersenI fllQQOl ) 



Party 


Poll 1 




Poll 2 




A 


416 


(33.1%) 


268 


(38.9%) 


B 


45 


(3.6%) 


22 


(3.2%) 


C 


338 


(26.9%) 


160 


(23.2%) 


E 


13 


(1.0%) 


6 


(0.9%) 


F 


131 


(10.4%) 


66 


(9.6%) 


K 


18 


(1.4%) 


10 


(1.5%) 


M 


47 


(3.7%) 


16 


(2.3%) 


Q 


20 


(1.6%) 


8 


(1.2%) 


V 


129 


(10.3%) 


92 


(13.4%) 


Y 


22 


(1.8%) 


9 


(1.3%) 


Z 


76 


(6.1%) 


32 


(4.6%) 


All 


1255 


(100.0%) 


689 


(100.0%) 



Table 4: The model of homogeneous proportions for Table [3] 



Party 


Poll 1 




Poll 2 




A 


441.6 


(35.2%) 


242.4 


(35.2%) 


B 


43.3 


(3.4%) 


23.7 


(3.4%) 


C 


321.5 


(25.6%) 


176.5 


(25.6%) 


E 


12.3 


(1.0%) 


6.7 


(1.0%) 


F 


127.2 


(10.1%) 


69.8 


(10.1%) 


K 


18.1 


(1.4%) 


9.9 


(1.4%) 


M 


40.7 


(3.2%) 


22.3 


(3.2%) 


Q 


18.1 


(1.4%) 


9.9 


(1.4%) 


V 


142.7 


(11.4%) 


78.3 


(11.4%) 


Y 


20.0 


(1.6%) 


11.0 


(1.6%) 


Z 


69.7 


(5.6%) 


38.3 


(5.6%) 


All 


1255.0 


(100.0%) 


689.0 


(100.0%) 
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Table 5: Differences between tfie entries of Table |3] and the corresponding entries of Table H] 



Party 


Poll 1 


Poll 2 


A 


-25.6 


25.6 


B 


1.7 


-1.7 


C 


16.5 


-16.5 


E 


0.7 


-0.7 


F 


3.8 


-3.8 


K 


-0.1 


0.1 


M 


6.3 


-6.3 


Q 


1.9 


-1.9 


V 


-13.7 


13.7 


Y 


2.0 


-2.0 


Z 


6.3 


-6.3 


All 


0.0 


0.0 



Table 6: The entries of Table [5] divided by the square roots of the corresponding entries of 
Table S 



Party 


Poll 1 


Poll 2 


A 


-1.2 


1.6 


B 


0.3 


-0.4 


C 


0.9 


-1.2 


E 


0.2 


-0.3 


F 


0.3 


-0.5 


K 


-0.0 


0.0 


M 


1.0 


-1.3 


Q 


0.5 


-0.6 


V 


-1.1 


1.5 


Y 


0.4 


-0.6 


Z 


0.8 


-1.0 



All 0.0 0.0 
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Table 7: Reasons for (or abs ence of) premature t ermination of the treatment of maniacal 
patients in three groups from lBowden et al. ( 1l994h (the three groups are those treated with 
divalproex, those treated with lithium, and those "treated" with a placebo) 



Reason 


Divalproex 




Lithium 




Placebo 




Lack of efficacy 


21 


(30.4%) 


12 


(33.3%) 


38 


(51.4%) 


Intolerance 


4 


(5.8%) 


4 


(11.1%) 


2 


(2.7%) 


Recovered 


3 


(4.3%) 


2 


(5.6%) 


2 


[2. 1 /o) 


Noncompliance 


1 


(1.4%) 


1 


(2.8%) 


3 


(4.1%) 


Another illness 





(0.0%) 


1 


(2.8%) 





(0.0%) 


Administration 


4 


(5.8%) 


2 


(5.6%) 


2 


(2.7%) 


Not terminated 


36 


(52.2%) 


14 


(38.9%) 


27 


(36.5%) 


All 


69 


(100.0%) 


36 


(100.0%) 


74 


(100.0%) 


Table 8: The mc 


del of homogeneous proportions for Table [7] 




Reason 


Divalproex 




Lithium 




Placebo 




Lack of efficacy 


27.4 


(39.7%) 


14.3 


(39.7%) 


29.4 


(39.7%) 


Intolerance 


3.9 


(5.6%) 


2.0 


(5.6%) 


4.1 


(5.6%) 


Recovered 


2.7 


(3.9%) 


1.4 


(3.9%) 


2.9 


(3.9%) 


Noncompliance 


1.9 


(2.8%) 


1.0 


(2.8%) 


2.1 


(2.8%) 


Another illness 


0.4 


(0.6%) 


0.2 


(0.6%) 


0.4 


(0.6%) 


Administration 


3.1 


(4.5%) 


1.6 


(4.5%) 


3.3 


(4.5%) 


Not terminated 


29.7 


(43.0%) 


15.5 


(43.0%) 


31.8 


(43.0%) 


All 


69.0 


(100.0%) 


36.0 


(100.0%) 


74.0 


(100.0%) 



Table 9: Differences between the entries of Table [7] and the corresponding entries of Table [8] 



Reason 


Divalproex 


Lithium 


Placebo 


Lack of efficacy 


-6.4 


-2.3 


8.6 


Intolerance 


0.1 


2.0 


-2.1 


Recovered 


0.3 


0.6 


-0.9 


Noncompliance 


-0.9 


0.0 


0.9 


Another illness 


-0.4 


0.8 


-0.4 


Administration 


0.9 


0.4 


-1.3 


Not terminated 


6.3 


-1.5 


-4.8 


All 


0.0 


0.0 


0.0 
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Table 10: The entries of Table |H] divided by the square roots of the corresponding entries of 
Table E 



Reason Divalproex Lithium Placebo 



Lack of efficacy 


-1.2 


-0.6 


1.6 


Intolerance 


0.1 


1.4 


-1.0 


Recovered 


0.2 


0.5 


-0.5 


Noncompliance 


-0.7 


0.0 


0.6 


Another illness 


-0.6 


1.8 


-0.6 


Administration 


0.5 


0.3 


-0.7 


Not terminated 


1.2 


-0.4 


-0.9 


All 


0.0 


0.0 


0.0 



Table 11: Results for the 2012 Republican U.S. presidential nomination, from a CBS News 
poll of November 6-10, 2011 (released November 11, 2011) and from a Pew Research Center 
poll of November 9-11, 2011 (released November 17, 2011), as reconstructed from percentages 
rounded to the nearest whole numbers (the original counts were not reported) for Republican 
primary voters 



Candidate 


CBS 




Pew 




Michele Bachmann 


15 


(4.6%) 


21 


(5.1%) 


Herman Cain 


69 


(21.2%) 


103 


(25.0%) 


Newt Gingrich 


57 


(17.5%) 


66 


(16.0%) 


Jon Huntsman 


4 


(1.2%) 


4 


(1.0%) 


Ron Paul 


19 


(5.8%) 


33 


(8.0%) 


Rick Perry 


31 


(9.5%) 


37 


(9.0%) 


Mitt Romney 


57 


(17.5%) 


91 


(22.1%) 


Rick Santorum 


8 


(2.5%) 


8 


(1.9%) 


Do not know 


65 


(20.0%) 


49 


(11.9%) 


All 


325 


(100.0%) 


412 


(100.0%) 
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Table 12: The model of homogeneous proportions for Table [TT] 



Candidate 


CBS 




Pew 




Michele Bachmann 


15.9 


(4.9%) 


20.1 


(4.9%) 


Herman Cain 


75.8 


(23.3%) 


96.2 


(23.3%) 


Newt Gingrich 


54.2 


(16.7%) 


68.8 


(16.7%) 


Jon Huntsman 


3.5 


(1.1%) 


4.5 


(1.1%) 


Ron Paul 


22.9 


(7.1%) 


29.1 


(7.1%) 


Rick Perry 


30.0 


(9.2%) 


38.0 


(9.2%) 


Mitt Romney 


65.3 


(20.1%) 


82.7 


(20.1%) 


Rick Santorum 


7.1 


(2.2%) 


8.9 


(2.2%) 


Do not know 


50.3 


(15.5%) 


63.7 


(15.5%) 


All 


325.0 


(100.0%) 


412.0 


(100.0%) 



Table 13: Differences between the entries of Table [TT] and the corresponding entries of 
Table [12] 



Candidate 


CBS 


Pew 


Michele Bachmann 


-0.9 


0.9 


Herman Cain 


-6.8 


6.8 


Newt Gingrich 


2.8 


-2.8 


Jon Huntsman 


0.5 


-0.5 


Ron Paul 


-3.9 


3.9 


Rick Perry 


1.0 


-1.0 


Mitt Romney 


-8.3 


8.3 


Rick Santorum 


0.9 


-0.9 


Do not know 


14.7 


-14.7 


All 


0.0 


0.0 
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Table 14: The entries of Table [13] divided by the square roots of the corresponding entries 
of Tabled 



Candidate 


CBS 


Pew 


Michele Bachmann 


-0.2 


0.2 


Herman Cain 


-0.8 


0.7 


Newt Gingrich 


0.4 


-0.3 


Jon Huntsman 


0.3 


-0.2 


Ron Paul 


-0.8 


0.7 


Rick Perry 


0.2 


-0.2 


Mitt Romney 


-1.0 


0.9 


Rick Santorum 


0.4 


-0.3 


Do not know 


2.1 


-1.8 


All 


0.0 


0.0 



Table 15: Reactions to prior treatment with lithium (when tr eated before with lithium) of 
maniacal patients in three groups from iBowden et al. fll994f ) (the three groups are those 



treated with divalproex, those treated with lithium, and those "treated" with a placebo) 
Reaction Divalproex Lithium Placebo 



Effective and 
tolerated 

Effective but 
not tolerated 

Ineffective but 
tolerated 

Ineffective and 
not tolerated 

No prior lithium 
treatment 



22 (31.9%) 16 (44.4%) 19 (25.7%) 

7 (10.1%) (0.0%) 6 (8.1%) 

19 (27.5%) 11 (30.6%) 31 (41.9%) 

6 (8.7%) 4 (11.1%) 5 (6.8%) 

15 (21.7%) 5 (13.9%) 13 (17.6%) 



All 69 (100.0%) 36 (100.0%) 74 (100.0%) 
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Reaction 



Table 16: The model of homogeneous proportions for Table [15] 
Divalproex Lithium Placebo 



Effective and 
tolerated 

Effective but 
not tolerated 

Ineffective but 
tolerated 

Ineffective and 
not tolerated 



22.0 (31.8%) 11.5 (31.8%) 23.6 (31.8%) 

5.0 (7.3%) 2.6 (7.3%) 5.4 (7.3%) 

23.5 (34.1%) 12.3 (34.1%) 25.2 (34.1%) 

5.8 (8.4%) 3.0 (8.4%) 6.2 (8.4%) 



No prior lithium ^2.7 (18.4%) 6.6 (18.4%) 13.6 (18.4%) 

treatment 



All 69.0 (100.0%) 36.0 (100.0%) 74.0 (100.0%) 



Table 17: Differences between the entries of Table [15] and the corresponding entries of 
Table M 



Reaction 


Divalproex 


Lithium 


Placebo 


Effective and tolerated 


0.0 


4.5 


-4.6 


Effective but not tolerated 


2.0 


-2.6 


0.6 


Ineffective but tolerated 


-4.5 


-1.3 


5.8 


Ineffective and not tolerated 


0.2 


1.0 


-1.2 


No prior lithium treatment 


2.3 


-1.6 


-0.6 



All 0.0 0.0 0.0 
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Table 18: The entries of Table [T7] divided by the square roots of the corresponding entries 
of Table M 



Reaction 



Divalproex Lithium Placebo 



Effective and tolerated 
Effective but not tolerated 
Ineffective but tolerated 
Ineffective and not tolerated 
No prior lithium treatment 



0.0 1.3 -0.9 

0.9 -1.6 0.3 

-0.9 -0.4 1.2 

0.1 0.6 -0.5 

0.6 -0.6 -0.2 



All 



0.0 0.0 0.0 
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